Skip to main content

Wide Crawl started March 2011

Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.

What’s in the data set:

Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069

The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.

However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.

We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available “warts and all” for people to experiment with. We have also done some further analysis of the content.

If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you’re hoping to do with it. We may not be able to say “yes” to all requests, since we’re just figuring out whether this is a good idea, but everyone will be considered.

8,528
RESULTS
rss


PART OF
Worldwide Web Crawls
Internet Archive Web Crawls
Media Type
8,528
web
Year
8,528
2011
Topics & Subjects
8,528
crawldata
Collection
Creator
8,528
internet archive
SHOW DETAILS
up-solid down-solid
eye
Title
Date Archived
Creator
Wide Crawl started March 2011
web
eye 456,112
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl415.us.archive.org:wide from Sun Aug 14 03:13:23 PDT 2011 to Sat Aug 13 22:20:18 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 275,098
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl410.us.archive.org:wide from Fri May 13 21:19:40 PDT 2011 to Fri May 13 22:10:12 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 273,352
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl448.us.archive.org:argov from Wed Aug 24 01:32:00 PDT 2011 to Thu Sep 1 12:02:59 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 250,876
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl423.us.archive.org:wide from Wed Apr 27 01:17:43 PDT 2011 to Tue Apr 26 20:39:17 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 236,141
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl420.us.archive.org:wide from Tue Apr 26 17:43:13 PDT 2011 to Tue Apr 26 12:42:46 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 227,575
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl416.us.archive.org:wide from Thu Jun 23 21:11:43 PDT 2011 to Thu Jun 23 14:53:15 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 213,664
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl426.us.archive.org:wide from Wed May 11 12:22:46 PDT 2011 to Wed May 11 08:39:50 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 213,274
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl417.us.archive.org:wide from Sat Apr 30 16:45:54 PDT 2011 to Sat Apr 30 21:10:35 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 192,936
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl426.us.archive.org:wide from Sun May 1 01:24:09 PDT 2011 to Sat Apr 30 22:56:48 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 189,038
favorite 0
comment 0
Internet Archive crawldata from Friendster Blogs Crawl, captured by crawl439.us.archive.org:friendster from Thu Jun 23 01:41:07 PDT 2011 to Wed Jun 22 19:53:32 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 187,263
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl421.us.archive.org:wide from Sat Jul 16 07:24:18 PDT 2011 to Sat Jul 16 00:57:31 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 179,035
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl422.us.archive.org:wide from Fri Apr 29 00:28:27 PDT 2011 to Thu Apr 28 20:15:25 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 169,198
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl421.us.archive.org:wide from Tue Apr 26 01:41:22 PDT 2011 to Mon Apr 25 20:29:54 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 166,977
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl421.us.archive.org:wide from Thu Oct 6 05:28:56 PDT 2011 to Thu Oct 6 00:58:42 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 164,991
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl416.us.archive.org:wide from Sun Jul 3 19:45:18 PDT 2011 to Sun Jul 3 13:40:14 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 160,820
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl426.us.archive.org:wide from Sat Apr 30 08:13:45 PDT 2011 to Sat Apr 30 06:46:45 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 160,109
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl410.us.archive.org:wide from Thu Apr 21 18:29:23 PDT 2011 to Thu Apr 21 17:57:54 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 160,097
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl424.us.archive.org:wide from Fri Apr 29 18:33:24 PDT 2011 to Fri Apr 29 14:42:40 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 158,394
favorite 0
comment 0
Internet Archive crawldata from Friendster Blogs Crawl, captured by crawl439.us.archive.org:friendster from Thu Jun 23 03:05:40 PDT 2011 to Wed Jun 22 20:25:57 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 157,941
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl418.us.archive.org:wide from Thu May 12 19:16:19 PDT 2011 to Thu May 12 17:08:02 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 157,273
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl422.us.archive.org:wide from Tue May 10 22:21:14 PDT 2011 to Thu May 12 10:52:29 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 156,485
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl425.us.archive.org:wide from Fri Apr 29 18:34:30 PDT 2011 to Fri Apr 29 14:28:25 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 155,277
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl448.us.archive.org:argov from Wed Aug 24 22:48:30 PDT 2011 to Thu Sep 1 12:13:09 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 154,655
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl425.us.archive.org:wide from Fri Jul 22 05:35:21 PDT 2011 to Thu Jul 21 23:36:33 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 146,526
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl429.us.archive.org:wide from Sat Apr 30 12:31:45 PDT 2011 to Sat Apr 30 11:48:18 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 144,379
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl426.us.archive.org:wide from Sun May 1 05:56:48 PDT 2011 to Sun May 1 03:35:51 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 143,286
favorite 0
comment 0
Internet Archive crawldata from Friendster Blogs Crawl, captured by crawl439.us.archive.org:friendster from Fri Jun 24 05:16:06 PDT 2011 to Thu Jun 23 22:45:10 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 141,299
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl419.us.archive.org:wide from Thu May 12 20:32:45 PDT 2011 to Thu May 12 16:18:38 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 138,218
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl425.us.archive.org:wide from Thu Jul 21 18:13:35 PDT 2011 to Thu Jul 21 12:04:47 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 137,425
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl338.us.archive.org:wide from Wed Mar 9 00:51:25 PST 2011 to Tue Mar 8 19:52:22 PST 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 136,627
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl411.us.archive.org:wide from Mon Jun 6 22:08:59 PDT 2011 to Mon Jun 6 16:02:13 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 128,763
favorite 0
comment 0
Internet Archive crawldata from Friendster Blogs Crawl, captured by crawl439.us.archive.org:friendster from Fri Jun 24 07:39:04 PDT 2011 to Fri Jun 24 01:20:52 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 128,087
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl427.us.archive.org:wide from Sat Aug 27 21:31:42 PDT 2011 to Sat Aug 27 22:07:22 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 128,034
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl416.us.archive.org:wide from Thu Jun 23 20:04:13 PDT 2011 to Thu Jun 23 14:00:57 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 125,389
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl428.us.archive.org:wide from Wed May 11 05:57:13 PDT 2011 to Wed May 11 02:09:02 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 122,488
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl413.us.archive.org:wide from Thu Apr 21 18:53:32 PDT 2011 to Fri Apr 22 04:21:25 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 122,035
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl415.us.archive.org:wide from Mon Aug 8 22:37:24 PDT 2011 to Mon Aug 8 19:58:57 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 118,947
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl416.us.archive.org:wide from Sat Apr 30 16:46:07 PDT 2011 to Sat Apr 30 15:17:39 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 116,381
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl425.us.archive.org:wide from Wed Jun 15 01:13:49 PDT 2011 to Tue Jun 14 20:47:44 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 115,882
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl413.us.archive.org:wide from Wed Jul 13 12:01:00 UTC 2011 to Wed Jul 13 12:57:54 UTC 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 115,867
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl418.us.archive.org:wide from Fri May 13 00:44:55 PDT 2011 to Fri May 13 00:41:56 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 115,574
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl427.us.archive.org:wide from Sat Apr 30 08:14:44 PDT 2011 to Sat Apr 30 06:19:55 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 114,255
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl425.us.archive.org:wide from Wed Jul 27 22:51:50 PDT 2011 to Wed Jul 27 16:59:32 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 113,791
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl425.us.archive.org:wide from Tue Jul 26 18:06:41 PDT 2011 to Tue Jul 26 12:55:43 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 113,112
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl415.us.archive.org:wide from Wed Apr 20 01:03:39 PDT 2011 to Thu Apr 21 22:48:08 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 112,599
favorite 0
comment 0
Internet Archive crawldata from Friendster Blogs Crawl, captured by crawl439.us.archive.org:friendster from Fri Jun 24 09:08:29 PDT 2011 to Fri Jun 24 02:59:41 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 112,254
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl414.us.archive.org:wide from Sat Apr 23 00:19:37 PDT 2011 to Sat Apr 23 04:18:58 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 111,377
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl421.us.archive.org:wide from Tue Oct 4 22:00:11 PDT 2011 to Tue Oct 4 16:07:45 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 111,219
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl420.us.archive.org:wide from Wed Apr 27 07:58:43 PDT 2011 to Tue Jun 14 13:40:31 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 110,590
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl423.us.archive.org:wide from Sat Jul 16 06:47:12 PDT 2011 to Sat Jul 16 00:46:51 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 109,887
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl416.us.archive.org:wide from Fri Jun 24 01:53:49 PDT 2011 to Thu Jun 23 19:38:20 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 108,726
favorite 0
comment 0
Internet Archive crawldata from Friendster Blogs Crawl, captured by crawl439.us.archive.org:friendster from Fri Jun 24 04:41:37 PDT 2011 to Thu Jun 23 22:14:12 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 107,612
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl411.us.archive.org:wide from Tue Apr 19 23:14:47 PDT 2011 to Thu Apr 21 11:32:30 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 106,871
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl428.us.archive.org:wide from Wed May 11 15:59:57 PDT 2011 to Wed May 11 13:06:59 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 106,223
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl421.us.archive.org:wide from Fri Apr 29 00:28:01 PDT 2011 to Fri Apr 29 01:47:45 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 106,158
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl422.us.archive.org:wide from Fri Apr 29 03:28:27 PDT 2011 to Thu Apr 28 21:48:35 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 105,790
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl426.us.archive.org:wide from Tue Jul 26 15:28:09 PDT 2011 to Tue Jul 26 09:53:32 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 105,400
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl413.us.archive.org:wide from Sun Jul 3 05:22:17 PDT 2011 to Sat Jul 2 23:10:34 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 105,394
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl411.us.archive.org:wide from Tue Jun 14 06:14:59 PDT 2011 to Tue Jun 14 00:23:44 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 105,000
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl417.us.archive.org:wide from Sun May 1 07:36:45 PDT 2011 to Sun May 1 03:33:21 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 103,927
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl426.us.archive.org:wide from Sat Apr 30 18:51:58 PDT 2011 to Sat Apr 30 17:48:25 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 103,102
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl448.us.archive.org:argov from Wed Aug 24 23:49:34 PDT 2011 to Thu Sep 1 17:17:29 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 102,374
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl411.us.archive.org:wide from Sat Aug 27 10:49:07 PDT 2011 to Sat Aug 27 05:40:30 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 101,883
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl423.us.archive.org:wide from Sat Jul 16 09:12:32 PDT 2011 to Sat Jul 16 03:34:11 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 100,606
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl424.us.archive.org:wide from Sat Apr 30 00:32:31 PDT 2011 to Fri Apr 29 19:53:45 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 98,846
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl417.us.archive.org:wide from Tue Aug 9 22:27:01 PDT 2011 to Tue Aug 9 20:55:46 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 98,584
favorite 0
comment 0
Internet Archive crawldata from Friendster Blogs Crawl, captured by crawl439.us.archive.org:friendster from Thu Jun 23 16:46:29 PDT 2011 to Thu Jun 23 10:34:07 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 97,089
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl423.us.archive.org:wide from Wed Jun 15 03:06:30 PDT 2011 to Tue Jun 14 21:53:14 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 96,564
favorite 0
comment 0
Internet Archive crawldata from Friendster Blogs Crawl, captured by crawl439.us.archive.org:friendster from Thu Jun 23 03:26:43 PDT 2011 to Wed Jun 22 20:46:48 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 95,647
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl420.us.archive.org:wide from Tue May 10 06:59:01 PDT 2011 to Tue May 10 02:36:49 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 95,614
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl418.us.archive.org:wide from Fri May 13 15:51:53 PDT 2011 to Fri May 13 19:08:43 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 95,416
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl412.us.archive.org:wide from Tue Sep 27 13:48:57 PDT 2011 to Tue Sep 27 07:46:21 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 94,944
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl421.us.archive.org:wide from Thu Oct 6 02:51:02 PDT 2011 to Wed Oct 5 21:45:35 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 94,703
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl413.us.archive.org:wide from Tue Jul 26 18:26:40 PDT 2011 to Tue Jul 26 12:04:03 PDT 2011.
Topic: crawldata
Wide Crawl started March 2011
web
eye 94,593
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl413.us.archive.org:wide from Sat Sep 24 16:30:43 PDT 2011 to Sat Sep 24 10:21:46 PDT 2011.
Topic: crawldata